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ABSIBACT 

The appropriateness of the National Board of Medical 
Examiners Part I examination was determined as an internal evaluation 
of second year students at a mianestern mediGal school. leviev of the 
June 1974 examination by 37 faculty revealed that 85 percent of the 
items reflected the content of the second year curriculum with 
information necessary to answer 60 percent provided in the second 
year teaching* Two--thirds of the items were first year 
content-prelate d and 50 percent could be answered on the basis of 
first year material. The school *s faculty used the study data in 
making policy-^cha^ging deciaiona, (luthor/BC) 
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ABSTRACT 



Externa 1 Exam i na 1 1 ons for 
The National Board Part I 

Warcia Wi le^ Ph.D, 
Case Western Reserve University 

This study examined the appropriateness of , the National Board of Hedjcal 
Examiners Part I exsmination as an internal evaluation of second year 
students at a midwestern rnedical school • Review of the June 197^ examinatfon 
by 37 faculty revealed that 8S percent of the items reflected the content 
of the second year curriculum with information necessary to answer 60 
percent provided In the second year teaching. Two^thirds of the items 
were first year content^related and 50 percent could be answered on Che 
basis of first year material , The school ' s facul ty used the study data 
in makiffig pol Icy^changing decisions. 



I nternal Eval uationi 
Examination as a Case 
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The orlgJnal purpose of the National Board of Med ical Examiners (NBME) 

was to develop certification examinations for the practice of medicine which 

would be aGcepted by state licensing boards, \ Tttree required eKaminatlons 

for certification were subsequently created, and all states except Arkansas 

2 

and Louisiana award licenses to diplomates of the NBME, Two of the NBME 
examinations (Part I and Part II) can be admintstered to undergraduate 
medical students, A majority of American medical schools requires its 
students to take these ex^^minat ions and uses the results for promotion and/or 
graduation. In the 1 975^ 1 976 academi c year , 90 of 119 medical schools in 
the United States and 1 of 16 medical schools in Canada were reported to 
require either or both Part I and Part IL^ This requirement has expandad 
the role of the NBHE to include the evaluation of undergraduate medical 
education programs*^ 

Although the NBME examinations have been used to measure the effect I veness 
of medical school curricula^ few studies have been reported regarding the 
congruence of these examinations with an individual school's curriculum. 
The NBME conducted a study in 1968 to determine the relevance of the Part I 
examination to current curricula of Unl ted States and Canadian medical 
schools in order to Improve future tests. ^ Basic science and cUnical science 
faculty from' 103 medical schools reviewed approximately 1800 I terns from 
SIX basic science subtests of recent Part I examinations* A yes-no response 
was requested as to each item's appropriateness for evaluating student 
Information in the basic science courses of the reviewer's scfhoolj^and Its 
reflecting essentia} knowledge which should require a mini mum pass level* 
Ninety percent of the items were considered to represent knowledge appropriate 
for the basic sciences^ and 65 percent of the items were determined to 
contafn essential content. : 

Medical school faculty at the University of Minnesota participated In 



an Investigation to assess the relationship between a NBME Part I examination 
and the school's basic science curriculum.^ Each coordinator m six basic 
science departments and selected faculty colleagues reviewed the conterjt 
area subtest of his discipUne of the Junm ]$7) exanrinat Iojit, The 

extent of current currfcsLilar emphasis was Judged for each item, and recommeodat ions 
for future teaching emphasis were offered. The results indicated a consistency 
of most of the items from this one examination with the curricular content. 
The reviewers accepted the content of the NBME eKami nat ion as a standard by which 
to evaluate their curriculum. They also identi fied areas for add i t iona 1 teaching 
activity. 

This paper describes a study in which the Part I examination was 
reviewed in terms of its relationship to the curriculum of a particular 
medical school and the parformarice of first and second year medical students. 
Case Western Reserve University (CWRU) School of Medicine has required 
Its students to take Part j and Part M of the INBME examinatfons since the 
introduction of the Interdepartmental system of teaching in 1952,7 jn this 
curriculum faculty from several disciplines taught content related to a 
specific organ system through the organizational unit of the subject committee. 
The cufriculum was divided into three phases: Phases 1 and 2, comprfsir^g two and 
one-half years of pre-cllnical curriculum; and Phase 3, eneompassing one and 
one-^half years of clinical activities, initially^ students took Pn^^t I 
at the completion of their basic science activities in the third year, and 
Part M at the end of the fourth year. Both examinations were considered 
to be external assessments with scores recorded but not used for promotion 
purposes. The faculty recogniied that the fntegrated subject committee teaching 
differed markedly from traditional medical school curricula, and did want 
to determine if the students cou 1 d qua li fy on a national certification . 
examination. With the curricular revision of 1968 came a change in the use 
off t Part I examination. The bas ic science curriculum was shortened to 



two years and the students v/ere required to take Part i at the end of the 
second year (Phase 2) as the second year comprehensive. Thus, this examination 
became an internal as wel T as an externa 1 eva 1 uac ion tool. 

In 1970 the NBME changed its policy to permit itudents to take Part I 
at the end of the first year, 8 Four first year CWRU students took Part I 
In 1971s while kO percent of the first year students took this examination in 
197^- During this period approximately 56 percent passed. These students 
entered the secortd year of medical schoo! having passed the comprehensive 
examination VHiithGut having experienced the Phase 2 currlculuffl. Since a 
consfderable proportion of the students could pass the^ Part I examination, 
the appropriateness of this examination as an internal assessment for the 
secorid year was questioned. 

This concern about the NBME Part I as ah evaluative initrument 
resulted in a pilot study conducted by the author in 197^- Eighteen faculty 
membars from 12 of the 13 second year subject cofmittees eKamined the June 1973 
HBhE Part ! examination in raistfon to the content and teaching activities in 
Phase 2t These reviewers considered approximately 75 percent of the items 
to be relevant to the Phase 2 curriculum and i nformat i on suf f i c i ent to 
answer 53 percent of the items was Included in the sacond year teaching. 

The Subcommittee on Intellectual Environment and Evaluation of the School's 
Committee on Medical Education endorsed an expanded study^ conducted in 
May 1975j which was designed to answer the following questions.* 

1, To what extent does the NBHE Part I examination correspond to 
the content of the first year (Phase I) and the second year (Phase 2) 
curriculum of tha CWRU School of Medicine? 

2, To what extent does the teaching in the Phase ! and Phase 2 subject 
committees emphasize information necessary to answer the items in the NBME 
Part I examination? 

. 3- What is Che level of performance of Phasa 1 and Phase 2 CWRU medical . 
students on all items reviewed? -6 / 



Method 

The Sevan subtests (anatomy, behavioral sciences, biochein^strys micro- 
biology, pathology, pharmacology, and physfology) of the NBME Part I 
examination administered In June 197^, were reviewed by 37 subject committee 
chairmen and teaching faculty representing the 7 Phase 1 and 13 Phase 2 
subject committeas* Each person Judged every Item as relevant or not 
relevant to the content of his particular subject committee. If the item 
were content^related the reviewer was asked to indicate one of the following: 

1. If mformation necessary to answer the item was taught In the subject 
committee which he represented* 

2, If information necessary to answer the item was not taught in 
his subject committee, 

3- If the reviewer did not know if information necessary to answer the 
item was taught in his subject committee. 

Additionany, the reviewers were encouraged to identify those items 
which were ambiguous, trivial, or had an incorrect answer indicated. They 
were also requested to specify the /Jreas taught In their subject committee 
teaching which were not tested by this particular NBME examination* 

This examination was taken by 121 second year students (class of 1976) 
and 16 first year students (class of 1977)* An item analysis of the 
performance of both the second y^ar and first year CWRU students which 
reported the percentage of^corre^ct answers for the local group and the national 
reference group was obtained from the NBME* 

There was a total of 1029 Items in the total test, but k] were deleted 
by the NBME because of the Itrms- poor performance. The Individual faculty 
reviews were summarized to generate a final Judgment for each of the 988 
Items.. If at least one facul ty member considered the I tern to be relevant, 
i t was categorized as relevant. If at least one reviewer indicated the 
relevant subject was taught, the item was classified as being taught. 



In all cases the intent was to include items as related to the CWRU 
curriculum, not to exclude them. Each item was assigned to one of four 
categories^ not relevanti relevant and taught, relevant and not taught^ 
and relevant and don*t know if taught. An analysis was completed For Phase 
1 items, Phase 2 items, and combined Phase land Phase 2 items* This was 
done for the total test and each of the seven subtests. Total test and 
subtest mean scores obtained by first year and second year students were 
calculated for both Phase 1 items^and Phase 2 items in the four categories , 
and comparisons were made between the first year and second year students' 
performance* 

Results : Content Relevance and Content Taught 

Table 1 presents the number of items in the total test which were 
assigned to each eategdry. The 15 Phase 1 reviewers considered 32 percent 
of the 988 Items unrelated to the Phase 1 curriculum and 68 percent related 
to the content of at least one Phase 1 subject conmittee. Information 
necessary to answer 51 percent of the I terns was taught In the first year. 
In the Judgments of the 22 Phase 2 reviewerSj almost 15 percent of the items 
were not related to the Phase 2 eurricuium, and 85 percent were relevant to 
the content of at least one Phase 2 subject committee. Sixty percent of 
the items could be answered based on instryx:±I^ri provided In the second year. 
When the reviews of the Phase land Phase 2 facu I ty were combined, the 
results indicated that 2// percent of the items were not related to either 
Phase^ While 12*2 percent were relatpd to the content of at least one 
Phase I or Phase 2 subject committee, the information necessary to 
answer these items was not taught i n the curriculum of the first two years. 
Eighty-five percent of the items were related to the content of at least 
one Phase 1 or Phase 2 subject committeej. and the informatian necessary 
to answer these i terns was taught in one or more subject commi ttees. 

The percent of Items In each of the seven subtests related and notj 
related to the combined Phase 1 and Phase 2 curricu lum was determined. These 



Table 1 



Class! f icatton of NBME Part 1 (June 197^) Items 



Category 



Phase I 



f\o, of items not relevant 315 (31*91) 

No. of Items relevant and taught 501 (50,71) 

No. of items relevant and not taught 118 (11.91) 

No. of items relevant and donH know 5^ ( 5*5i) 



Phase 2 

590 (60.01) 
Zkl (25.01) 
9 ( KOI) 



Phase I and Phase 2 

27 ( 2.71) 
121 (12,21) 
840 (85.01) 




data are summarized in Table 2. In this particular NBME examination, 98.6 
percent of the physiology I terns were relevant to the content of the first 
two year curriculum and had teaching emphasisV Approximately 90 percent of 
the anatomy, biochemistry and pathology Items were related to and taught 
In the first or second year. Only 7^ percent of the pharmacology items 
and 62 percent of the behavioral sciences Items were content-related 
and had information taught in the Phase 1 and Phase 2 curricula. 
Phase 1 related and taught items ranged from 1 percent for 
behavioral sciences to 82 percent for micrnblology , Information related to 
approximately 75 percent of biochemfstry and physiology items, and 
approximately 33 percent of pathology and pharmacology I terns was taught. - ^ 
More items were related to the content of the Phase 2 subject committees and 
were emphasized In the teaching. The greatest representation was in physiology^ 
followed by pathology and anatomy* The least was in microbiology. 

The faculty rev i ewers from Phase 2 were asked to list topics or areas 
taught in their particular subject commi ttees which had not been covered 
in the NBME examination. Among the ident i fied topics were* head and neck 
anatomy and neuroanatomy^ chi Id development and adult psychopathology ^ 
valvular disorders^ esophageal and intestinal motility and associated 
pathophysiology, and mechanisms underlying development of acidosis in renal 
disease. The reviewers also designated those items which were ambiguous, 
trivial, or poor quality. Many of the Items considered to be '^poor'- 
questions by the CWRU faculty a priori were included in the ^1 items deleted 
by the National Board on the basis of performance of the national group, 
Resul ts : Student Performance 

Mean scores were calculated for first year student and second year 
student performance in the fol lowing categories* Phase 1-relevant and 
taught items, Phase l-relevant and not taught items, Phaie 1-not relevant items, 
Phase 2-relevant and taught items, Phase 2-relevant and not taught items,. _ 



.Table 2 



= Percent 


of Content-Relevant 


and Taught 1 terns 






Phase 1 


Phase 2 


Phase 1 and Phase 2 


Ana tonny 


46.2 


77.6 


90.9 


Behav i ora 1 Sc 1 ences 


0.7 


61.5 


61.5 


Biochemi stry 


77.1 


kk.k 


90.3 


Microbiology 


82. i» 


17.6 


86.8 


Pathology 


37.7 


79.5 




Pharmacology 


33.6 


53.6 


7^.3 


Physiology 


75.7 


81.3 


98.6 



11 



ERIC 



and Phase 2-not relevint items, A t-test for independent samples was 
used to make comparisons between differences in mean scores of the Phase 1 
and Phase 2 students in each classification. 

The results of these analyses are summarized in Table 3, Second year 
students achieved statistically significant higher scores than did first 
year students In three categories^ 1) items relevant to the content of 
the second year with teaching emphasis In the second year subject committees, 
2) Items related to the first year curriculum^ but not taught In the first 
year; and 3) items not relevant to the first year curriculum. The higher 
performance by the second year students In the second category could 
be explained by the fact that Items related to Phase 1 but not having 
teaching emphasis In the first year may have had currlcular emphasis In 
the second year. In the third category, many of the items not related 
to Phase 1 could have been related to the Phase 2 content, ^ 

Independent sample t-test comparisons were made between the performance 
.on content-related items which were taught and content-related items which 
were not taught. Both first and second year students obtained statistical ly 
significant higher scores on Items whose content was related to the Phase 
I curriculum and taught than on Items whose content was related to the Phase 

1 curriculum but not taught. Second year students achieved statistical ly 
significant higher scores on Phase 2 related and taught Items than on Phase 

2 related items whose content was not taught/ There was no difference 
in" the mean scores of first year students on items related to Phase 2^ 
whether with or without teaching emphasis. Tables 4 and present these finding 
D I scuss ion 

The information generated by the faculty review of the NBME Part I 
examination revealed thaL much of what was asked was not taught in the - 
second year CWRU curriculum. Only 60 percent of the questions could be 
answered by information provided in the Phase 2 subject commi ttees * There were 



.Table 3 





Performanee of 

- ■ . . - . . . 


Second Year Students 
Ft rst Year Students • 


■Compared to 
-Total Test 


Performance of 




. f ■ , 


• Category 




No. of 
1 terns 








Phase 1 -relevant and 


taught Items 


501 


-0.87 : 


N.S. 




Phase 1 -relevant and 


not taught items 


118 


2.53 


< .01 




Phase 1-not relevant 


i terns 


3' 5 


5.81 


< .001 




Phase 2-Televant and 


taught I terns 


590 


5.81 


<.00! 




Phase 2-relevant and 


not taught items 


243 


-0.43 


N.S. 




Phase 2-not relevant 


Items 


1 46 


-0.64 


N.S. 
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Parformance on Phase 1 Relevant Items and 
Taught Compared to Phaie 1 Re 1 eva nt Items 
and Not Taught 

Group t p 

Firit Year Students 
Second Year Students 



7-^7 <.001 
4. 58 <. 00] 
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Table 5 



Pe r f o rma nc© on Phase 2 Re 1 a van t I t ems and Taught 
Com pared to Phase 2 Relevant ItemsandNot Taught 



G roup ' t 2^ 

Firs t Year Students -1.92 N,S, 

Second Year Students 3,20 ^,01 




significant areas which were taught rn the second year, but wtre not 
^testedj^n this eKaml nation. Yet the Part I examination was being used as 
ithm second year comprehensive. This disparity between the content of the 
'National Board exam f nation and the CWRU curriculum Influenced many students 
to; spend a great deal of time preparing for the eKamlnation ind were diverted 
from participating in the curriculum duriny the last two or three months' 
of the second year, 

A considerable number of first year itudents was able to pass the Part 
I examination at the end of Phase I , The faculty review Indicated that 50 
percent of the questions could be answered on the basis of the first year 
course material. Students passing the National Board at the end of tfia ^ 
first year were also passing the second year comprehensive before the 
second year began! 

In terms of performance on the National Board examination , second year 
students scored higher than f I rst year students on i tems whose content was 
related to and taught In Phase 2 subject commi ttees ; on items related to 
PhaseO^but not taught; and on Phase 1-non-related items. Both first 
and second year students ach I eved higher scores on i tems. whos^jcon ten jLv^__ 
related to and taught In thei r respect ive subject committees than on items 
which were content related, but not taught. Teaching made a difference! 
Faculty Decision 

, The data from this study were submitted to the Subcommittee on fntellectual 
Environment and Evaluation, The Subcomm i ttee's charge was to review the 
current methods of evaluating student performance, to assess their influence 
on the Intel lectualcl Imate of the school ^ and to recommend changes In policy 

which would improve the overal 1 evaluation process and the Intel 1 actual 

" i ....... ■ " 

climate* The resul ts of this study were incjuded In the Subcommittee's : 

presentation concerning the National Board Part 1 at a facul ty^student -- 

colloquy held in June 1975v The discussion and feedback generated from 
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this meet ing rasui ted in two racommendatlbns by the Subcommittees ; 

1. That the. facul ty no longer require CWRU medical students to take 
the NBME Part 1 examination, 

2, That a required Phase 2 comprehensive examination created by the 
faculty be administered at the compl tt ton of Phase 2, 

These recommendations were approved by the Comml ttee on Medical 
Education and were iubsequently presented to the General Faculty* After 
extensive discussions at two meetings in October 1975, the General Facul ty 
voted to accept the recommendations; Starting with the class of 1979^ CWRU 
second year itudents will take an internally developed examination at the 
end of Phase 2, and the NBME Part I examination wlil not be required, 
Concl us ion 

The review of the NBME Part i by the CWRU faculty had an impact In 
decision-making and determination of poncy. Although the National Board 
examinations had been required for more than two decades, there had not 
been a systematized review of these exam f nations in relation to the CWRU 
curriculum. As a consequence of this study ^ the facul ty recogn i zed that the 
NBME Part I examination was appropriate for 1 icensure^ but not appropriate 
for internaJ student evaluation. The faculty decision emphasized that 
internal evaluation of CWRU students should be the responsibi 1 ity of the 
school - s facul ty and not left to an outside agency* ^ 
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